Sentiment per President

Let’s understand how many “positive” and “negative” words are used by each president across all their SONA speeches and create a metric called “sentiment” which is simply the total number of positive words minus the total number of negative words. We then adjust for the total number of words used from the lexicon in the “sentiment_score” metric.

Of the 3 presidents that have made more than 1 SONA speech, Zuma has the highest sentiment score, followed by Mbeki and then Mandela. Zuma’s sentiment score is nearly double Mandela’s. It’s interesting that the current President, Ramaphosa, has the second highest sentiment score, not far behind Zuma and only slightly ahead of Mbeki.

What are the 10 positive words most frequently used by each president?

De Klerk’s most used words were “freedom”, “peaceful” and “support” and at least 2 of these 3 come up in all the president’s most used words. Mandela’s most used words include “progress”, “improve”, “reconciliation” and “commitment” which are all words indicating repair and a move towards something better. Mbeki uses many of the same words but also introduces “empowerment” which is a word carried through by Zuma and Ramaphosa, and “success” which is carried through by Zuma. This is likely due to the fact that Black Economic Empowerment (BEE) was introduced under Mbeki and was a policy carried through by Zuma and Ramaphosa. In addition, these words suggest progress in the move towards repair or something better, first spoken about by Mandela. Ramaphosa also introduces the words “confidence”, “effectively”, “enhance” and “efficient”, which are words commonly seen in a business context and have not shown up in any other SA president’s top 10 most frequently used words in a SONA since 1994.

Which of the positive words most frequently used are common across presidents?

Common positive words across post 1994 presidents include: “freedom”, “regard”, “support”, “improve” and “progress”. Words introduced by Mandela and unique to his speeches are: “restructuring”, “reconciliation”, “committment”, “contribution” and “succeed”. Mbeki introduces the words “empowerment”, “comprehensive”, “integrated” and “improving” into the top words used and this is unique to his speeches. Zuma uses the words “success”, “reform” and “pleased” frequently and other presidents do not. Ramaphosa introduces the words “significant”, “productive”, “confidence” and “effectively” which have not yet been seen in the any other SA president’s top 10 most frequently used words in a SONA since 1994.

What are the 10 negative words most used by each president?

## Joining, by = "word"

Common negative words pre 1994 include: “concerns”/“concern”/“concerned”, “unconstitutional”, “illusion”, “hopeless”, “disagree”, “deprive”, “conflict”, and “boycott”.

Common negative words post 1994 include: “corruption”, “crime”/“criminal”, “poverty”/“poor”, “inequality”, “issue”/“issues” and “crisis”.

A negative word introduced by and unique to Mandela’s top 10 is “struggle”. Mbeki is the only president with the word “racism” in his top 10 negative words. Motlanthe has “conflict” in his top 10 which no other president does. Zuma has “rail” which likely refers to the railway system and does have negative connotations for South Africa. Both Zuma and Ramaphosa use the word “difficult” a lot. Ramaphosa introduces the word “expropriation” into the top 10 for the first time amongst his peers.

How many of the negative words most used were used by each president?

The interpretation is much the same as before. Note the clear separation between the top 10 negative words used pre and post 1994 elections, indicative of the pre and post apartheid regimes.

What proportion of words used are positive vs negative?

The 2 vertical black lines are drawn at 60% and 70% positivity rates. In the majority of years, SONA speeches fall within this range of positivity however a there are a few more negative speeches in earlier years and a few more positive speeches in later years.

Is negative sentiment increasing over time?

## 
## Call:
## glm(formula = freq ~ as.numeric(year), family = "binomial", data = subset(sentiments_relative, 
##     sentiment == "negative"))
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -0.21779  -0.05759   0.01224   0.07113   0.19692  
## 
## Coefficients:
##                   Estimate Std. Error z value Pr(>|z|)
## (Intercept)       25.88105  115.18655   0.225    0.822
## as.numeric(year)  -0.01316    0.05743  -0.229    0.819
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 0.39087  on 24  degrees of freedom
## Residual deviance: 0.33826  on 23  degrees of freedom
## AIC: 27.484
## 
## Number of Fisher Scoring iterations: 3

The slope is negative but the beta of the year variable is not significant so we cannot conclude that negative sentiment is decreasing over time.

Is postive sentiment increasing over time?

## 
## Call:
## glm(formula = freq ~ as.numeric(year), family = "binomial", data = subset(sentiments_relative, 
##     sentiment == "positive"))
## 
## Deviance Residuals: 
##      Min        1Q    Median        3Q       Max  
## -0.19692  -0.07113  -0.01224   0.05759   0.21779  
## 
## Coefficients:
##                   Estimate Std. Error z value Pr(>|z|)
## (Intercept)      -25.88105  115.18655  -0.225    0.822
## as.numeric(year)   0.01316    0.05743   0.229    0.819
## 
## (Dispersion parameter for binomial family taken to be 1)
## 
##     Null deviance: 0.39087  on 24  degrees of freedom
## Residual deviance: 0.33826  on 23  degrees of freedom
## AIC: 27.484
## 
## Number of Fisher Scoring iterations: 3

The slope is positive but the beta of the year variable is not significant so we cannot conclude that positive sentiment is increasing over time.

Is average sentiment increasing over time?

## 
## Call:
## glm(formula = avg_sentiment ~ as.numeric(year), family = "gaussian", 
##     data = sentiments_per_year)
## 
## Deviance Residuals: 
##     Min       1Q   Median       3Q      Max  
## -21.252   -7.877   -1.282    8.584   21.444  
## 
## Coefficients:
##                    Estimate Std. Error t value Pr(>|t|)  
## (Intercept)      -1369.9476   652.4104  -2.100   0.0460 *
## as.numeric(year)     0.6952     0.3253   2.137   0.0425 *
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## (Dispersion parameter for gaussian family taken to be 153.4216)
## 
##     Null deviance: 4536.4  on 26  degrees of freedom
## Residual deviance: 3835.5  on 25  degrees of freedom
## AIC: 216.44
## 
## Number of Fisher Scoring iterations: 2

The slope is positive and the beta of the year variable is significant at 1% so we can conclude that average sentiment is increasing over time.

But we need to be cautious with this interpretation because what could actually be going on here is that the “bing”" lexicon has more than double the number of negative words than positive words so this could be influencing the results and SONA speeches may in fact be more positive than they appear to be.

## 
## negative positive 
##     4782     2006

Distribution of “bing” Sentiment per President

Apart from the last 2 presidents, Ramaphosa and Zuma, the presidents are in time order. We can see that other than Motlanthe, the trend is an increasing average sentiment over time but at a decreasing rate. The interquartile range of Mbeki is smaller than Zuma’s which is smaller than Mandela’s.

Change in “bing” Sentiment over time

Average sentiment is the proportion of positive words out of all the words in the “bing” lexicon. Mandela shows a very erratic average sentiment, ranging from 0 to over 25. Mbeki and Zuma’s average sentiment mostly ranges between 25 and 50, with the exception of a few such as 2000, 2008, 2012, 2017.

Distribution of “afinn” Sentiment per President

The interpretation is much the same as with the “bing” lexicon in that the trend is an increasing average sentiment over time however Zuma’s median sentiment is lower than the general trend.

Change in “afinn” Sentiment over time

Mandela and Zuma show a wave-like pattern of sentiment. Mbeki shows an increasing and then decreasing pattern.

Checking for Negation in Bigrams

It just so happens that the 4 negation words are also stop words so have already been removed from the Bigrams and we need to add them back. This can be shown as follows:

##  [1] "not"     "not"     "not"     "no"      "no"      "no"      "never"  
##  [8] "never"   "without" "without"

Let’s redo the bigrams without removing stop words and see how many bigrams contain 1 of the negation words:

## # A tibble: 1 x 1
##       n
##   <int>
## 1   118

There are only 118 bigrams that contain negation words. Let’s look at a few examples:

## Selecting by id

Let’s see how many there are per president:

Given that there is such a low percentage of bigrams with negation words, it is not expected to significantly change the interpretation above and recoding the sentiments is not justified.

Step 1: Tokenisation, Remove Stop words and Document Term Matrix

Figure: Most popular terms

After tokenisation and removal of stop words, the top 20 most used terms across all of the SONA speeches are displayed. Unsurprisingly, “South Africa” is the most used term followed closely by “South African” and “South Africans” and “Local Government”. These terms do not add to our understanding of the topics and tend to confuse the topic modelling going forward. The removal of the terms allows for a cleaner interpretation. "Public service is then the most used term.

Step 2: Optimisation of k - the number of topics.

A pre-requisite of topic modelling is knowing the number of topics that each corpus may contain (i.e. the latent factor k). In some cases, this may be a fair assumption but without reading though each speech, how could one know how many different topics have been articulated in the SONA’s? Luckily, Murzintcev Nikita has published an R- package (ldatuning) that helps to optimise the number of topics (k) over three different measures. The measures used to determine the number of topics, are discussed in an RPubs paper which can found here: link and the following optimisation largely follows the accompanying vignette: [link] (https://cran.r-project.org/web/packages/ldatuning/vignettes/topics.html)

The following extract from the RPub paper gives a brief explanation of the methods used to optimise for k:

Extract from RPubs

"Arun2010: The measure is computed in terms of symmetric KL-Divergence of salient distributions that are derived from these matrix factor and is observed that the divergence values are higher for non-optimal number of topics (maximize)

CaoJuan2009: method of adaptively selecting the best LDA model based on density.(minimize)

Griffths: To evaluate the consequences of changing the number of topics T, used the Gibbs sampling algorithm to obtain samples from the posterior distribution over z at several choices of T(minimize)"

In addition to this, Nikita considers how the number of k may change over a validation or hold out sample. His term for this is“perplexity” which he defines as “[it] measures the log-likelihood of a held-out test set; Perplexity is a measurement of how well a probability distribution or probability model predicts a sample”

Below is an attempt to optimise for k and to check that the choice of k holds over an unseen data set.

From the above plot, the marginal benefit from adding another topic, stops at around 5-10 topics. In order to test this, the “perplexity” over a test sample for the document term matrix can be checked.

Figure: Perplexity Plot

As more topics are used, the perplexity of the training sample does decrease but that of the test sample increases from around 11 topics. The perplexity of the test sample seems to be minimised at around 5 topics.

The evidence from these two plots suggest that the optimal number topics sits at 5 topics.

Step 3: Latent Dirichlet allocation

For this assignment, Latent Dirichlet allocation (LDA) was used for the topic modelling. Other methods, such as a Latent Semantic Analysis (LSA) or Probabilistic Latent Semantic Analysis (pLSA) could have been used but LDA is useful due to the fact that it allows:

1. Each document within the corpus to be a mixture of topics
2. Each topic to be a mixture of bigrams
3. The topics are assumed to be drawn from Dirichlet distribution (i.e. not k different distributions as with pLSA) so there are fewer parameters to estimate and no need to estimate the probability that the corpus generates a specific document.

Step 4: Extracting insights

Understanding the topics via the bigrams

The beta matrix produced gives the probability of the topic producing that bigram (i.e. that the phrase is in reference to that topic.) From this measure, one can get a sense of what the character of the topic is. By using the most popular phrases in each topic, understanding of the flavour of each topic emerges. However, it must be kept in mind that terms can belong to more than one topic so applying some logic to get a theme or flavour should be done liberally.

Topic 1

From the display of popular terms, it can be determined that the topic one has a vague connection to “job creation” - this is the most common terms but is supported by other terms that have a high probability of being generated by this topic such as: + “world cup” + “national youth” + “infrastructure development”

These concepts all support the idea of the job creation as each of these will generate jobs for the country. But there is some noise in the topic for “address terms” i.e. honourable speaker or honourable chairperson. “Nelson Mandela” and “President Mandela” crop up too which suggests that alongside the job creation theme, there exists some of what can be termed “terms of endearment”

Topic 2

As with the previous topic, there are some random “terms of endearment” in this topic (i.e. “madam speaker”) but it is not as evident as in the first topic. This is to be expected as bigrams can be generated by more than one topic as each topic is a mixture of bigrams! The next four terms sum up the main themes for this topic:

• “Economic Empowerment”
• “Black Economic”
• “Justice System”
• “Criminal Justice”

This topic can be named as “Economy/ Criminal and Justice System”.

Topic 3

Despite the most popular terms being “United Nation” and “private sector”, there is a theme that is “developing”. As in development plan, resource development, national development and development programme etc. and thus, the topic is named “Development”.

Topic 4

Once again, there is a “term of endearment” in the popular terms (“fellow south” which is assumed short for “fellow South African’s” which is one of former President Zuma’s favourite phrases). With all the other terms combined, a theme of “Social Reform/ Regional and Municipal Government” takes shape.

Given that there is a possible trigram evident here, it may be worth exploring in future work.

Topic 5

“Public sector” and “private sector” are popular terms in topic 5. After consideration of the various other terms, of which some have cross over with other topics and discussion, the eventual name for this topic became “Public Sector Entities”

A different way of looking at this topic could be to investigate the biggest differential in terms between topics. For instance, using the log(base 2) ratio between topic 1 and topic 5, shows the terms that have the widest margin between the two topics (i.e. they are far more likely to be in topic 5 versus topic 1).

Figure: Comparison of biggest differential between Topic 1 and Topic 5 Bigram

For instance, “social programmes”, “human fulfilment”, “rights commission” are all generated in significantly larger proportions by Topic 5 compared to Topic 1 while “national social”, “training colleagues” and “sector unions” all exist with in Topic 1.

Given the naming of Topic 5 as “Public Sector Entities” and Topic 1 as “Job Creation/Terms of Endearment” these terms do seem to be grouped in line with expectation.

Understanding the mixture of topics within the sentence

The LDA model allows each of the sentence to be represented as a mixture of topics. The gamma matrix shows the document-topic probability for each sentence. i.e. the probability that each sentence is drawn from that topic. For instance, the following sentence sampled from random shows that it has a 0.905% probability of being drawn from topic 4 based on the use of the bigrams within it. The sentence appears to be talking about the water and the infrastructure around it. The label for topic of was “Social Reform/ Regional and Municipal Government” and this statement seems to be somewhat relevant to it.

Sample sentence showing topic probabilities

president	year	sentence	X1	X2	X3	X4	X5
Zuma	2010	yet, we still lose a lot of water through leaking pipes and inadequate infrastructure.	0.023575	0.023575	0.023575	0.9057001	0.023575

Using this method, the sentences can be roughly classified to a topic based on the probabilities (i.e. classify the sentence by the topic with the highest probability) and further analysis can be conducted.

(Note: the which.is.max breaks ties at random so where a sentence has equal probabilities, is will decide at random to which topic it gets assigned)

Figure: Probability of belonging to each Topic, by President

Consider the mixture of topics that each individual president covers during the SONA address. Despite the imbalance in the number of sentences said by each president, there seems to be a fairly standard shape to the topics discussed. The two exceptions to this are de Klerk and Zuma. All other presidents tend to send around 10-15% on topic 1 (“Job Creation/Terms of Endearment”) and the 15-20% on Topic 2 (“Economy/Criminal and Justice System”), Topic 3 (“Development”) and Topic 4 (“Social Reform/Regional and Municipal Government”) and the around another 10% on Topic 5 (“Public Sector Entities”). This trend means that it may be difficult for a supervised model to pick up difference in presidents based on the topic covered.

As stated, the only two presidents where this trend differs are President de Klerk and President Zuma. President de Klerk spend the majority of his time on Topic 1 (“Job Creation/Terms of Endearment”) followed by Topic 2 (“Economy/Criminal and Justice System”). Given the context around the time period, it may be unsurprising that “terms of endearment” and “criminal and Justice systems” come up since his speech would be littered with names of people and political parties as well as talking about past injustices.

President Zuma spends the majority of his speeches on Topic 4 (“Social Reform/Regional and Municipal Government”). Once again, given context that his terms as President was marked with service delivery strikes, two major droughts over a few different regions and discussions around and reform this may be unsurprisingly. And in fact, when the most popular terms from topic 4 is recalled (“fellow south”) is may even be predictable that this would be the most “talked about” topic for President Zuma. What is interesting that given the attention to the issues of State Capture that characterised Zuma’s presidency, his coverage of Topic 5 (“Public Sector Enterprises”) is much smaller than that of his peers.

A similar analysis can be taken over time.

Figure: Change in topic composition over time, per president

The graph shows that over time, topics 1 and 5 are the least discussed topics while topics 2,3 and 4 all get much the same airtime. There are a number of notable spikes/valleys:

• In 1996, Topic 2 (“Economy/Crime and Justice System”) spikes
  • The 1996 SONA was a few months ahead of the introduction of the new constitution as well as at the time of the start of the Truth and Reconciliation Commission. It could be suggested that these two ideas would drive up the topic in the SONA speech.
• In 2005, topic 1 (“Job Creations/Terms of Endearment”) dives while topic 4 (“Social Reform/Region and Municipal Government”) and Topic 2 (“Economy/Criminal and Justice System”) spike considerably.

• Mbeki’s term in presidency (1998 - 2008) was characterised by a rise in crime specifically in farm attacks as well as the HIV/AID epidemic and the start of the Black Economic Empowerment in 2005 which could attribute the spikes and drops of topics in 2005.

• In 2012, Topic 2 dives considerably (“Economy/Criminal and Justice System”).

• From various media reports, Zuma’s 2012 SONA speech largely covered the success of the government while skipping over future plans. It may be a reason while Topic 4 (“Social Reform/Regional and Municipal Government”) rises sharply.

Step 5: Using the topic to predict the president

One of the aims behind topic modelling is to reduce the dimensions of the data to allow for other techniques to be applied. In this instance, the aim was to reduce the SONA speeches to a collection of topics that would help predict which president was responsible for a sentence in the SONA speech. The assumption was that each president might have a unique set of topics or mixture of topics that could characterise their particular speech. However, there does not seem to be evidence of this. The matrix with the probability of each sentence belonging to a topic is used in the Neural Nets with Topic Modelling section and the results are discussed.

Cohen’s Kappa

The Kappa value tells you how much better your classifier is performing than simply guessing at random according to the frequency of each class.

“Cohen’s kappa is always less than or equal to 1. Values of 0 or less, indicate that the classifier is useless. There is no standardized way to interpret its values. Landis and Koch (1977) provide a way to characterize values. According to their scheme a value < 0 is indicating no agreement, 0-0.20 as slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1 as almost perfect agreement.” [Reference: Landis, J.R.; Koch, G.G. (1977). “The measurement of observer agreement for categorical data”. Biometrics 33 (1): 159-174]

The Kappa value is, 0.416 and this means we are doing better than random.

The accuracy is slightly better than bigger model with no-dropouts (0.581%) Just for the word-count model seems good enough. But this model does not consider how important each word is to its corpus. So we should consider a better model to try.

After the fourth iteration validation loss starts increasing which is a sign of overfitting.

Cohen’s Kappa

The accuracy is 0.6158 and the model starts overfitting after fourth epoch. So this is slightly better than the bag of words word-count model as we expected.

Embeddings

An embedding layer is a dimensionality reduction technique that attempts to encode the relationships between words in a sentence (input as a variable length integer array with padding) as a fixed-length floating point vector. An embedding has a tunable hyper-parameter for the number of latent factors to map every sentence on to, where each latent factor attempts to capture a semantic dimension of the sentence as a whole. Embeddings aim to capture the linear substructure of sentences through Euclidean distances between words in the n-dimensional unit hypercube, where n is the number of latent factors specified.

Embeddings can be trained on the corpus of sentences that comprise the dataset under investigation, however this can prove limiting if the there is a relatively small quantity of training data. An alternative approach is to re-use a previously trained embedding layer, such as the Glove embedding. This has the advantage of leveraging the results from a much larger, and theoretically more generic, dataset in an application of transfer learning. The SONA data has a large number of non-standard or foreign words included, however, which theoretically limits the applicability of pre-trained embeddings.

Chosen structure

We experimented with a number of different network topologies, and finally settled on the following:

• An embedding layer with 70 latent factors trained on the full word dictionary across all speeches.
• A dropout layer set to randomly exclude 50% of inputs from the previous layer on each iteration, to prevent overfitting.
• A convolutional layer with 50 filters, a kernel size of 3, and stride of 1
• A global max pooling layer, which assists in dimensionality reduction
• A fully connected dense layer with 128 nodes
• A dropout layer set to randomly exclude 50% of inputs from the previous layer on each iteration, to prevent overfitting.
• An activation layer using the “relu” function
• A fully connected dense layer with 6 nodes, to map to our output encoding
• An activation layer using the “softmax” function, to produce output consistent with our one-hot-encoding of presidents.

Cohen’s Kappa

With a Kappa value of 0.2036538, we are doing moderately better than a random classifier.

Chosen structure

We experimented with a number of different network topologies, and finally settled on the following:

• An embedding layer with 70 latent factors trained on the full word dictionary across all speeches.
• A dropout layer set to randomly exclude 50% of inputs from the previous layer on each iteration, to prevent overfitting.
• A convolutional layer with 50 filters, a kernel size of 3, and stride of 1
• A global max pooling layer, which assists in dimensionality reduction
• A fully connected dense layer with 128 nodes
• A dropout layer set to randomly exclude 50% of inputs from the previous layer on each iteration, to prevent overfitting.
• An activation layer using the “relu” function
• A fully connected dense layer with 6 nodes, to map to our output encoding
• An activation layer using the “softmax” function, to produce output consistent with our one-hot-encoding of presidents.

Results

RNN’s have been shown to achieve very good performance when applied to Natural Language Processing (NLP) of text, due to the similarities with how humans process language. Unfortunately our applied RNN did not surpass the performance of the other networks we attempted, despite many attempts at tuning. We suspect the sparsity of the dataset played a large role in this result, as it was replicated on both the balanced and unbalanced training data.